Analysis: Studying the Properties of Topic Models with Different Numbers of Topics

A number of researchers have suggested that one of the limitations of LDA is that it cannot identify how many topics are in a corpus, leaving this decision to the human user (Yau et al., 2014 and Suominen and Toianen, 2016). Indeed, there is no way to identify the "correct" number of topics in advance of building the topic model (Carter et al., 2016). If the user specifies too few topics for the model, then the topics will be too general and useless for exploratory analysis or information retrieval. By contrast, if the user specifies too many topics for the model, the topics will be too specific, or redundant, to be of use; also, too many topics makes the interpretation of the model unwieldy. Therefore, most users experiment with the number of topics and make qualitative evaluations about which number of topics is most useful (Chang et al., 2016). Ultimately the right choice about the number of topics is dependent upon the way in which the model is going to be used (Carter, et al., 2016). As such, the ratio of documents (n) in a corpus to topics (k) to be extracted from the corpus ranges widely. Just to provide a few examples:

Here, I analyze the properties of topic models each of which have a different number of topics:

  • model_25_topics: This model has 25 topics, is based on a noun-only corpus, and has the alpha value set to symmetric.
  • model_75_topics: This model has 25 topics, is based on a noun-only corpus, and has the alpha value set to symmetric.
  • model_150_topics: This model has 150 topics, is based on a noun-only corpus, and has the alpha value set to symmetric.

Set Up: Import Packages and Load Topic Models

In [4]:
from gensim import corpora, models, similarities
import pyLDAvis.gensim
import json
import spacy

path = '../noun_corpus/'

# load metadata for later use
with open('../data/doc2metadata.json', encoding='utf8', mode='r') as f:
    doc2metadata = json.load(f)
    
# load dictionary and corpus for the noun models
dictionary = corpora.Dictionary.load(path + 'noun_corpus.dict')
corpus = corpora.MmCorpus(path + 'noun_corpus.mm')

# load model_25_topics
model_25_topics = models.ldamodel.LdaModel.load(path + 'noun_25.model')

# load model_75_topics
model_75_topics = models.ldamodel.LdaModel.load(path + 'noun_75.model')

# load model_150_topics
model_150_topics = models.ldamodel.LdaModel.load(path + 'noun_150.model')

Topic Coherence Test

Topic Coherence: model_25_topics

In [10]:
model_25_viz = pyLDAvis.gensim.prepare(model_25_topics, corpus, dictionary)
pyLDAvis.display(model_25_viz)
Out[10]:

model_25_topics produced 4 topics which lack semantic or contextual coherence, 2 topics of mixed coherence, and 19 topics which are coherent. Therefore its topics are:

  • 16% junk topics
  • 8% mixed topics
  • 76% coherent topics

To illustrate what is meant by each category, consider the following examples:

Examples of junk topics:

  • Topic 4: essay, bible, john, commentary, theology, james, paul, old, hebrew, introduction
  • Topic 6: spirit, world, revelation, life, angel, enoch, christ, death, idea, philo, lord

Example of mixed topic:

  • Topic 10: job, verb, wisdom, meaning, phrase, proverbs, sense, clause, context, noun

Examples of coherent topics:

  • Topic 2 (narrative criticism): narrative, story, analysis, reader, structure, speech, context, reading, character, function
  • Topic 9 (bible versions): hebrew, lxx, aramaic, translation, mt, greek, version, reading, meaning, targum
  • Topic 11 (textual criticism): reading, manuscript, greek, edition, codex, variant, version, line, e, fragment
  • Topic 18 (Dead Sea Scrolls): qumran, scroll, jerusalem, city, sea, period, dead, site, palestine, wall
  • Topic 23 (Poetry): psalm, psalms, line, song, poem, poetry, prayer, unit, psalter

Topic Coherence: model_75_topics

In [23]:
noun_75_viz = pyLDAvis.gensim.prepare(noun_75, corpus, dictionary)
pyLDAvis.display(noun_75_viz)
Out[23]:

model_75_topics produced 13 topics which lack semantic or contextual coherence, 5 topics of mixed coherence, and 57 topics which are coherent. Therefore its topics are:

  • 17.3% junk topics
  • 6.6% mixed topics
  • 76% coherent topics

A number of the topics from the model_25_topics reappear in the model_75_topics. However, some topics, such as topic 9 from model_25_topics appear to be given more nuance in model_75_topics, for example:

  • topic 35 (bible versions 1): lxx, hebrew, greek, translation, translator, mt, version, septuagint, reading, bible
  • topic 45 (bible versions 2): version, syriac, mt, targum, reading, manuscript, old, peshita, edition, variant It would be possible to dismiss these two topics as redundant. However, topic 35 appears to emphasize Greek versions (lxx, greek, septuagint) whereas topic 45 emphasizes Hebrew (mt, targum) and Syriac (syriac, peshita).

model_75_topics also introduces many new coherent topics not found in model_25_topics, for example:

  • topic 27 (apocalyptic literature): revelation, enoch, angel, apocalypse, vision, son, messiah, heaven, baruch, judgment
  • topic 49 (gender): woman, gender, male, husband, mary, sex, wife, body, role, hair
  • topic 59 (midrash): scripture, torah, rabbi, midrash, canon, mishnah, talmud, neusner, mishna, rabbinic

Topic Coherence: model_150_topics

In [4]:
model_150_viz = pyLDAvis.gensim.prepare(model_150_topics, corpus, dictionary)
pyLDAvis.display(model_150_viz)
Out[4]:

model_150_topics produced 33 topics which lack semantic or contextual coherence, 8 topics of mixed coherence, and 109 topics which are coherent. Therefore its topics are:

  • 22% junk topics
  • 5% mixed topics
  • 73% coherent topics

The coherent topics found in the previous models are present in the model_150_topics, but a large number of other coherent topics are added, for example:

  • topic 78 (holiness Code): leviticus, p, numbers, holiness, code, priestly, offering, legislation, exodus, milgrom
  • topic 104 (song of songs): song, metaphor, love, songs, lover, odes, beloved, solomon, bride, poem
  • topic 112 (patristics): justin, marcion, eusebius, ireneaus, tertullian, papias, hippolytus, tesitmony, epiphanius, harnack

Topic Coherence: Brief Discussion

model_25_topics contained 76% coherent topics, model_75_topics contained 77% coherent topics and model_150_topics contained 73% coherent topics. So, relative to the number of topics in each model, the performance was similar. However, given the raw numbers, model_150_topics contains far more coherent topics than either of the other two models. This suggests that model_150_topics provides a more nuanced model of the corpus. Topics which did not register in the other models, such as topic 78 (holiness code) and topic 112 (patristics), are revealed in model_150_topics. The utility of having nuanced topics needs to be wighed against the difficulty of keeping track of so many topics while doing an exploritory analysis of a corpus; nuance comes at the cost of efficenty.

Clustering Test

In [4]:
def cluster_test(corpus, model):
    docs_with_1_topic = 0
    docs_with_multiple_topics = 0
    docs_with_no_topics = 0
    total_docs = 0
    for doc in corpus:
        topics = model.get_document_topics(doc, minimum_probability=0.20)
        total_docs += 1
        if len(topics) == 1:
            docs_with_1_topic += 1
        elif len(topics) > 1:
            docs_with_multiple_topics += 1
        else:
            docs_with_no_topics += 1
    print('Corpus assigned to a single topic:', (docs_with_1_topic / total_docs) * 100, '%')
    print('Corpus assigned to multiple topics:', (docs_with_multiple_topics / total_docs) * 100, '%')
    print('corpus assigned to no topics:', (docs_with_no_topics / total_docs) * 100, '%')

Clustering: model_25_topics

In [5]:
cluster_test(corpus, model_25_topics)
Corpus assigned to a single topic: 42.55455712451862 %
Corpus assigned to multiple topics: 55.96919127086007 %
corpus assigned to no topics: 1.4762516046213094 %

Clustering: model_25_topics

In [6]:
cluster_test(corpus, model_75_topics)
Corpus assigned to a single topic: 55.26315789473685 %
Corpus assigned to multiple topics: 30.14548566538297 %
corpus assigned to no topics: 14.591356439880187 %

Clustering: model_150_topics

In [7]:
cluster_test(corpus, model_150_topics)
Corpus assigned to a single topic: 54.04364569961489 %
Corpus assigned to multiple topics: 17.811296534017973 %
corpus assigned to no topics: 28.145057766367138 %

Clustering: Brief Discussion

model_25_topics outperforms the other two models in that it only left 1.47% of documents unassigned to a topic. By contrast, model_75_topics left 14.59% of documents assigned and model_150_topics left 28.14% of the documents unassigned. Additionaly, although model_25_topics assigned fewer documents to a single topic than the other two models, it assigned more far more documents to multiple topics, thus providing a more robust clustering system where a document may belong to more than one topic.

Information Retrieval Test

In [7]:
# build indicies for similarity quiries
index_25 = similarities.MatrixSimilarity(model_25_topics[corpus])  
index_75 = similarities.MatrixSimilarity(model_75_topics[corpus])  
index_150 = similarities.MatrixSimilarity(model_150_topics[corpus])

# define retrieval text
def retrieval_test(new_doc, lda, index):
    new_bow = dictionary.doc2bow(new_doc)  # change new document to bag of words representation
    new_vec = lda[new_bow]  # change new bag of words to a vector
    index.num_best = 10  # set index to generate 10 best results
    matches = (index[new_vec])
    scores = []
    for match in matches:
        score = (match[1])
        scores.append(score)
        score = str(score)
        key = 'doc_' + str(match[0])
        article_dict = doc2metadata[key]
        author = article_dict['author']
        title = article_dict['title']
        year = article_dict['pub_year']
        print(key + ': ' + author.title() + ' (' + year + '). ' + title.title() + '\n\tsimilarity score -> ' + score + '\n')
    average_score = sum(scores) / len(scores)
    print('*********************************')
    print("Average similarity score ->", average_score)
    
# set up nlp for new docs
nlp = spacy.load('en')
stop_words = spacy.en.STOPWORDS

def get_noun_lemmas(text):
    doc = nlp(text)
    tokens = [token for token in doc]
    noun_tokens = [token for token in tokens if token.tag_ == 'NN' or token.tag_ == 'NNP' or token.tag_ == 'NNS']
    noun_lemmas = [noun_token.lemma_ for noun_token in noun_tokens if noun_token.is_alpha]
    noun_lemmas = [noun_lemma for noun_lemma in noun_lemmas if noun_lemma not in stop_words]
    return noun_lemmas

# load and process Greene, N. E. (2017)
with open('../abstracts/greene.txt', encoding='utf8', mode='r') as f:
    text = f.read()
    greene = get_noun_lemmas(text)
    
#load and process Hollenback, G. M. (2017)
with open('../abstracts/hollenback.txt', encoding='utf8', mode='r') as f:
    text = f.read()
    hollenback = get_noun_lemmas(text)

# load and process Dinkler, M. B. (2017)
with open('../abstracts/dinkler.txt', encoding='utf8', mode='r') as f:
    text = f.read()
    dinkler = get_noun_lemmas(text)
/Users/msaxton/anaconda3/lib/python3.6/site-packages/gensim/models/ldamodel.py:495: RuntimeWarning: invalid value encountered in multiply
  gammad = self.alpha + expElogthetad * np.dot(cts / phinorm, expElogbetad.T)

Finding Articles Similar to Greene, N. E. (2017). Creation, destruction, and a Psalmist's plea: rethinking the poetic structure of Psalm 74.

Information Retrieval: model_25_topics

In [28]:
retrieval_test(greene, model_25_topics, index_25)
doc_3297: Muilenburg, James (1944). Psalm 47
	similarity score -> 0.9694860577583313

doc_4673: Globe, Alexander (1974). The Literary Structure And Unity Of The Song Of Deborah
	similarity score -> 0.9577165842056274

doc_2855: Jefferson, Helen Genevieve (1952). Psalm 93
	similarity score -> 0.9295371174812317

doc_5369: Gerstenberger, Erhard (1963). Review Of The Psalms In Israel'S Worship
	similarity score -> 0.9221556782722473

doc_3360: Montgomery, James A. (1945). Stanza-Formation In Hebrew Poetry
	similarity score -> 0.9174710512161255

doc_7288: Limburg, James (1997). Review Of  Jahwe Wird Kommen, Zu Herrschen Über Die Erde: Ps 90-110 Als Komposition 
	similarity score -> 0.9103174209594727

doc_8205: Waltke, Bruce K. (1991). Superscripts, Postcripts, Or Both
	similarity score -> 0.904938817024231

doc_8304: Gladson, Jerry A. (1993). Review Of The Song Of Songs: A Commentary On The Book Of Canticles Or The Song Of Songs
	similarity score -> 0.8977451920509338

doc_2231: Shea, William H. (1986). Chiasmus And The Structure Of David'S Lament
	similarity score -> 0.890748143196106

doc_3092: Hyatt, J. Philip (1950). Review Of The Psalms Translated And Interpreted In The Light Of Hebrew Life And Worship
	similarity score -> 0.8892325162887573

*********************************
Average similarity score -> 0.9189348578453064

Information Retrieval: model_75_topics

In [29]:
retrieval_test(greene, model_75_topics, index_75)
doc_9217: Briggs, Charles A. (1899). An Inductive Study Of Selah
	similarity score -> 0.8951220512390137

doc_1411: Berry, George R. (1914). The Titles Of The Psalms
	similarity score -> 0.8929077386856079

doc_804: Peters, John P. (1921). Another Folk Song
	similarity score -> 0.8910684585571289

doc_757: Peters, John P. (1916). Ritual In The Psalms
	similarity score -> 0.890021800994873

doc_2855: Jefferson, Helen Genevieve (1952). Psalm 93
	similarity score -> 0.8780856728553772

doc_123: Armstrong, Ryan M. (2012). Psalms Dwelling Together In Unity: The Placement Of Psalms 133 And 134 In Two Different Psalms Collections
	similarity score -> 0.8754755258560181

doc_8205: Waltke, Bruce K. (1991). Superscripts, Postcripts, Or Both
	similarity score -> 0.8719521164894104

doc_9314: Peters, John P. (1910). Notes On Some Ritual Uses Of The Psalms
	similarity score -> 0.8674021363258362

doc_2970: Liebreich, Leon J. (1955). The Songs Of Ascents And The Priestly Blessing
	similarity score -> 0.8623942732810974

doc_5418: Buss, Martin J. (1963). The Psalms Of Asaph And Korah
	similarity score -> 0.8536677360534668

*********************************
Average similarity score -> 0.877809751033783

Information Retrieval: model_150_topics

In [30]:
retrieval_test(greene, model_150_topics, index_150)
doc_7176: Malchow, Bruce V. (1997). Review Of Psalm 102 Im Kontext Des Vierten Psalmenbuches
	similarity score -> 0.7946841716766357

doc_7288: Limburg, James (1997). Review Of  Jahwe Wird Kommen, Zu Herrschen Über Die Erde: Ps 90-110 Als Komposition 
	similarity score -> 0.7869090437889099

doc_2855: Jefferson, Helen Genevieve (1952). Psalm 93
	similarity score -> 0.7821860313415527

doc_123: Armstrong, Ryan M. (2012). Psalms Dwelling Together In Unity: The Placement Of Psalms 133 And 134 In Two Different Psalms Collections
	similarity score -> 0.7742323279380798

doc_9217: Briggs, Charles A. (1899). An Inductive Study Of Selah
	similarity score -> 0.7709280848503113

doc_1411: Berry, George R. (1914). The Titles Of The Psalms
	similarity score -> 0.7587219476699829

doc_4324: Buss, Martin J. (1970). Review Of Studien Zur Formgeschichte Von Hymnus Und Danklied In Israel
	similarity score -> 0.7457179427146912

doc_5418: Buss, Martin J. (1963). The Psalms Of Asaph And Korah
	similarity score -> 0.7401379346847534

doc_7286: Miller, Patrick D. (1997). Review Of Die Komposition Des Psalters: Ein Formgeschichtlicher Ansatz
	similarity score -> 0.7374092936515808

doc_8205: Waltke, Bruce K. (1991). Superscripts, Postcripts, Or Both
	similarity score -> 0.722113311290741

*********************************
Average similarity score -> 0.7613040089607239

Brief Discussion: Finding articles similar to Greene, N. E. (2017). Creation, destruction, and a Psalmist's plea: rethinking the poetic structure of Psalm 74.

The average similarity score for the top ten results for the first information retrieval task are as follows:

  • model_25_topics: average similarity score -> 91.8%
  • model_75_topics: average similarity score -> 87.7%
  • model_150_topics: average similarity score -> 76.1%

Two documents from the corpus were matches with the Greene article in all three models:

  • doc_2855: Jefferson, Helen Genevieve (1952). Psalm 93
  • doc_8205: Waltke, Bruce K. (1991). Superscripts, Postcripts, Or Both

doc_2855 shows up as the 3rd highest match in model_25_topics (similarity score of 92.9%) and model_150_topics (similarity score of 78.2%), but as the 7th highest match in the model_75_topics (similarity score of 87.1%). doc_8205 shows up as the 7th highest match in model_25_topics (similarity score of 90.4%) and model_75_topics (similarity score of 87.1%), but as the 10th highest match in model_150_topics (similarity score of 72.2%).

Finding Articles Similar to Hollenback, G. M. (2017). Who is doing what to whom revisited: Another look at Leviticus 18:22 and 20:13.

Information Retrieval: model_25_topics

In [31]:
retrieval_test(hollenback, model_25_topics, index_25)
doc_5543: Weiss, David Halivni (1962).  A Note On <Rle>אשר לא ארשה<Pdf> 
	similarity score -> 0.8941407203674316

doc_672: Ginzberg, Louis (1922). Some Observations On The Attitude Of The Synagogue Towards The Apocalyptic-Eschatological Writings
	similarity score -> 0.8482827544212341

doc_2271: Peterj. Haas (1986). Review Of Support For The Poor In The Mishnaic Law Of Agriculture: Tractate Peah
	similarity score -> 0.8410084247589111

doc_7969: Morrow, William (1993). Review Of Property And The Family In Biblical Law
	similarity score -> 0.8349608778953552

doc_1594: Avery-Peck, Alan J. (1986). Review Of A History Of The Mishnaic Law Of Damages: Part 2: Baba Mesia: Translation And Explanation
	similarity score -> 0.8196205496788025

doc_3876: Goodblatt, David (1973). Review Of Theft In Early Jewish Law
	similarity score -> 0.8144832849502563

doc_8207: Kraemer, David (1991). The Formation Of Rabbinic Canon: Authority And Boundaries
	similarity score -> 0.8011728525161743

doc_2682: Lieberman, Saul (1952). The Discipline In The So-Called Dead Sea Manual Of Discipline
	similarity score -> 0.7950565814971924

doc_5715: Silberman, Lou H. (1963). Review Of The Book Of Asseverations
	similarity score -> 0.7941067218780518

doc_6038: Garber, Zev (1977). Review Of Essays In Jewish And Comparative Legal History
	similarity score -> 0.7867633104324341

*********************************
Average similarity score -> 0.8229596078395843

Information Retrieval: model_75_topics

In [32]:
retrieval_test(hollenback, model_75_topics, index_75)
doc_8995: Martin, Troy W. (2004). Paul'S Argument From Nature For The Veil In 1 Corinthians 11:13-15: A Testicle Instead Of A Head Covering
	similarity score -> 0.8026383519172668

doc_463: Cosgrove, Charles H. (2005). A Woman'S Unbound Hair In The Greco-Roman World, With Special Reference To The Story Of The "Sinful Woman" In Luke 7:36-50
	similarity score -> 0.7838690280914307

doc_8719: Burrus, Virginia (1999). Review Of Early Christian Women And Pagan Opinion: The Power Of The Hysterical Woman
	similarity score -> 0.709067702293396

doc_1851: Kraemer, Ross S. (1985). Review Of In Memory Of Her: A Feminist Theological Reconstruction Of Christian Origins
	similarity score -> 0.7029363512992859

doc_143: Townsley, Jeramy (2011). Paul, The Goddess Religions, And Queer Sects: Romans 1:23—28
	similarity score -> 0.702646791934967

doc_1974: Trible, Phyllis (1987). Review Of The Israelite Woman: Social Role And Literary Type In Biblical Narrative
	similarity score -> 0.6896610260009766

doc_284: Lemos, T. M. (2006). Shame And Mutilation Of Enemies In The Hebrew Bible
	similarity score -> 0.6890443563461304

doc_3940: Bailey, John A. (1970). Initiation And The Primal Woman In Gilgamesh And Genesis 2-3
	similarity score -> 0.6796943545341492

doc_7634: D'Angelo, Mary Rose (1990). Women In Luke-Acts: A Redactional View
	similarity score -> 0.6690856218338013

doc_8757: Walsh, Jerome T. (2001). Leviticus 18:22 And 20:13: Who Is Doing What To Whom?
	similarity score -> 0.6592174768447876

*********************************
Average similarity score -> 0.7087861061096191

Information Retrieval: model_150_topics

In [33]:
retrieval_test(hollenback, model_150_topics, index_150)
doc_7348: Pardee, Dennis (1997). Review Of  Gottes Himmlischer Thronrat: Hintergrund Und Bedeutung Von Sôd Jhwh Im Alten Testament 
	similarity score -> 0.6823307275772095

doc_1830: Buth, Randall (1985).  Luke 19:31-34, Mishnaic Hebrew, And Bible Translation: Is Κύριοι Τοῦ Πώλου Singular? 
	similarity score -> 0.5826741456985474

doc_638: Baab, Otto J. (1933). A Theory Of Two Translators For The Greek Genesis
	similarity score -> 0.5576117634773254

doc_3029: Rabinowitz, Jacob (1954). A Note On Isa 46:4
	similarity score -> 0.5556772947311401

doc_4536: Thomas, J. D. (1972). The Greek Text Of Tobit
	similarity score -> 0.5543149709701538

doc_387: Büchner, Dirk (2010). 'Eξιλ
	similarity score -> 0.5503463745117188

doc_1313: Byington, Steven T. (1920).  <Rle>חשֹד<Pdf> 
	similarity score -> 0.5493355989456177

doc_3087: Daube, David (1950).  Jesus And The Samaritan Woman: The Meaning Of Συγχράομαι 
	similarity score -> 0.5430412888526917

doc_548: Burrows, Millar (1930). The Original Language Of The Gospel Of John
	similarity score -> 0.5419495105743408

doc_9271: Ciampa, Roy E. (2011). "Examined The Scriptures"? The Meaning Of Άναϰρίνοντες Τάς Γραφάς In Acts 17:11
	similarity score -> 0.5403329730033875

*********************************
Average similarity score -> 0.5657614648342133

Brief Discussion: Finding articles similar to Hollenback, G. M. (2017). Who is doing what to whom revisited: Another Look at Leviticus 18:22 and 20:13.

The average similarity score for the top ten results for the second information retrieval task are as follows:

  • model_25_topics: average similarity score -> 82.2%
  • model_75_topics: average similarity score -> 70.8%
  • model_150_topics: average similarity score -> 56.5%

model_25_topics returned results having to do with biblical law and rabbinic interpretation with an average similarity score of 82.2%. model_75_topics returned results that focus primarily on issues of gender and sexuality with an average similarity score of 70.8%. Finally, model_150_topics returned results focusing on translation issues with an average similarity score of 56.5%. Clearly, each model understands this article differently. All three themes--law, gender/sexuality, and translation issues-- are present in the article, so in a sense each model is useful. However, and interestingly, none of these models returned the article to which the present one is a response: Walsh, J.T. (2001). Leviticus 18:22 and 20:13: Who is Doing What to Whom? Journal of Biblical Literature, 120, 201-9.

Finding Articles Similar to Dinkler, M. B. (2017). Building character on the road to Emmaus: Lukan characterization in contemporary literary perspective.

Information Retrieval: model_25_topics

In [34]:
retrieval_test(dinkler, model_25_topics, index_25)
doc_6960: Collins, Adela Yarbro (1994). Review Of Teaching With Authority: Miracles And Christology In The Gospel Of Mark
	similarity score -> 0.9818892478942871

doc_8418: Chilton, Bruce (1993). Review Of Die Letzten Tage Jesu: Markus Und Johannes, Ihre Traditionen Und Die Historische Frage, Band 1
	similarity score -> 0.9816499948501587

doc_7626: Smith, D. Moody (1990). Review Of The Fourth Gospel And Its Predecessor: From Narrative Source To Present Gospel
	similarity score -> 0.9804149866104126

doc_8262: Malbon, Elizabeth Struthers (1988). Review Of Sense And Absence: Structure And Suspension In The Ending Of Mark'S Gospel
	similarity score -> 0.9802979826927185

doc_7786: Segovia, Fernando F. (1989). Review Of The Humanity Of Jesus In The Fourth Gospel
	similarity score -> 0.9789214730262756

doc_7819: Collins, Adela Yarbro (1993). Review Of  "Eine Neue Lehre In Vollmacht": Die Streit- Und Schulgespräche Des Markus-Evangeliums 
	similarity score -> 0.9773369431495667

doc_5909: Donahue, John R. (1980). Review Of Mark'S Treatment Of The Jewish Leaders
	similarity score -> 0.9766092896461487

doc_8712: Brodie, Thomas L. (1999). Review Of The Discipleship Paradigm: Readers And Anonymous Characters In The Fourth Gospel
	similarity score -> 0.9748271703720093

doc_7428: Senior, Donald (1996). Review Of  Die Älteste Bericht Über Den Tod Jesu: Literarische Analyse Und Historische Kritik Der Passionsdarstellungen Der Evangelien 
	similarity score -> 0.9739975929260254

doc_8227: Black, C. Clifton (1991). Review Of Faith As A Theme In Mark'S Narrative
	similarity score -> 0.9721850156784058

*********************************
Average similarity score -> 0.9778129696846009

Information Retrieval: model_75_topics

In [35]:
retrieval_test(dinkler, model_75_topics, index_75)
doc_8712: Brodie, Thomas L. (1999). Review Of The Discipleship Paradigm: Readers And Anonymous Characters In The Fourth Gospel
	similarity score -> 0.8676302433013916

doc_8158: Tyson, Joseph B. (1988). Review Of The Lukan Voice: Confusion And Irony In The Gospel Of Luke
	similarity score -> 0.8573089241981506

doc_264: Ahearne-Kroll, Stephen P. (2010). Audience Inclusion And Exclusion As Rhetorical Technique In The Gospel Of Mark
	similarity score -> 0.8564040064811707

doc_7866: Lincoln, Andrew T. (1989). The Promise And The Failure: Mark 16:7, 8
	similarity score -> 0.8410413265228271

doc_7796: Malbon, Elizabeth Struthers (1993). Echoes And Foreshadowings In Mark 4-8 Reading And Rereading
	similarity score -> 0.8389743566513062

doc_1952: Praeder, Susan Marie (1984). Review Of Mark As Story: An Introduction To The Narrative Of A Gospel
	similarity score -> 0.8336173892021179

doc_7865: Malbon, Elizabeth Struthers (1989). The Jewish Leaders In The Gospel Of Mark: A Literary Study Of Marcan Characterization
	similarity score -> 0.8162114024162292

doc_8420: Cassidy, Richard J. (1993). Review Of Conflict In Luke: Jesus, Authorities, Disciples
	similarity score -> 0.8124485611915588

doc_9260: Iverson, Kelly R. (2011). A Centurion'S "Confession": A Performance-Critical Analysis Of Mark 15:39
	similarity score -> 0.8065377473831177

doc_6706: Boomershine, Thomas E. (1981). Mark 16:8 And The Apostolic Commission
	similarity score -> 0.7936846017837524

*********************************
Average similarity score -> 0.8323858559131623

Information Retrieval: model_150_topics

In [37]:
retrieval_test(dinkler, model_150_topics, index_150)
doc_8712: Brodie, Thomas L. (1999). Review Of The Discipleship Paradigm: Readers And Anonymous Characters In The Fourth Gospel
	similarity score -> 0.8001492023468018

doc_8420: Cassidy, Richard J. (1993). Review Of Conflict In Luke: Jesus, Authorities, Disciples
	similarity score -> 0.7993659973144531

doc_7865: Malbon, Elizabeth Struthers (1989). The Jewish Leaders In The Gospel Of Mark: A Literary Study Of Marcan Characterization
	similarity score -> 0.7983871698379517

doc_7897: Powell, Mark Allan (1990). The Religious Leaders In Luke: A Literary-Critical Study
	similarity score -> 0.7918199300765991

doc_312: Sylva, Dennis (2006). Review Of Dialogue And Drama: Elements Of Greek Tragedy In The Fourth Gospel
	similarity score -> 0.7893494367599487

doc_7746: Swartley, Willard M. (1991). Review Of Galilee, Jesus And The Gospels: Literary Approaches And Historical Investigations
	similarity score -> 0.7863094806671143

doc_7796: Malbon, Elizabeth Struthers (1993). Echoes And Foreshadowings In Mark 4-8 Reading And Rereading
	similarity score -> 0.7700948119163513

doc_8446: Bautch, Richard J. (2004). Review Of Pontius Pilate: Portraits Of A Roman Governor
	similarity score -> 0.7639981508255005

doc_8083: Anderson, Janice Capel (1992). Review Of Matthew'S Missionary Discourse: A Literary Critical Analysis
	similarity score -> 0.7592097520828247

doc_7819: Collins, Adela Yarbro (1993). Review Of  "Eine Neue Lehre In Vollmacht": Die Streit- Und Schulgespräche Des Markus-Evangeliums 
	similarity score -> 0.7500901222229004

*********************************
Average similarity score -> 0.7808774054050446

Brief Discussion: Finding Articles Similar to Dinkler, M. B. (2017). Building character on the road to Emmaus: Lukan characterization in contemporary literary perspective.

The average similarity score for the top ten results for the third information retrieval task are as follows:

  • model_25_topics: average similarity score -> 97.7%
  • model_75_topics: average similarity score -> 83.2%
  • model_150_topics: average similarity score -> 78.0%

Each topic model retrieved docuuments dealing with the gospels which on a general level is fitting for this article. There is one document from the corpus which was retireved by all three models:

  • doc_8712: Brodie, Thomas L. (1999). Review Of The Discipleship Paradigm: Readers And Anonymous Characters In The Fourth Gospel

The model_25_topics ranked this as the 8th highest match (similarity score of 97.4%) whereas both model_75_topics and model_150_topics ranked this as the 1st highest match(similarity scores of 86.7% and 80.0% respectivley). It may seem strange that these two models ranked this document as the highest match insofar as it is about the Gospel of John but the query article was about the Gospel of Luke, but the nuance provided by these models are picking up the themes of literary charactiorizartion.

Summary Discussion

The number of topics assigned to a topic model has a significant effect on the properties of the topic model. On the one hand, assigning more topics to a model allows the model to group patterns of word co-occurrence in more nuanced ways. This allows for more specific, but possibly redundant topics. On the other hand, since each topic is distributed as a probability across each document (however low the probability may be), and since those probabilities must total 1.0, it follows that the more topics that are assigned, the lower, on average, those probabilities will be. How does this affect topic coherence, clustering, and information retrieval?

The three models examined here each produced approximately the same percentage of junk topics, mixed topics, and coherent topics. However, in terms of raw numbers, model_150_topics produced far more coherent topics than did the other models. The coherent topics from model_150_topics are also more specific than those provided by the other models. However, this larger number of coherent topics, and the level of specificity, comes at a cost; namely, more junk topics, potential redundancy in topics, and the difficulty of keeping track of so many topics (to say nothing of the additional computational time). So while the larger number of topics produces a larger amount of coherent topics, and is therefore useful for exploratory analysis of a large corpus, a researcher may decide on a middle ground between a small number of topics and a large number of topics as a way of balancing cost and benefit.

model_25_topics was able to cluster nearly the entire corpus of the JBL, whereas the other two models fell far short of that goal. Of course, such clustering is only useful it the topics are coherent. Assigning a smaller number of topics to a model allows the model to cluster more documents with a topic at a higher threshold (like the 20% threshold used here) because the values in the probability distribution will be higher. Therefore, a smaller number of topics is desirable if the goal of the topic model is cluster documents.

As far as information retrieval is concerned, the smaller the number of topics assigned to the model, the higher the similarity score will be for matching documents; but this does not necessarily mean that model_25_topics retrieved more relevant articles than did model_150_topics. Rather, since model_25_topics's are less specific, it may return only general matches at a higher similarity score. The topics from model_150_topics are more specific and will therefore return matches at a lower similarity score. In other words, model_25_topics may fail to return matches which would be useful to the researcher. model_75_topcis may be a nice compromise between general matches at a higher similarity score and specific matches at a lower similarity score. On the whole, model_75_topics, achieved higher similarity scores than did model_150_topics, but it also has a higher level of nuance than model_25_topics by virtue of having a higher level of nuance.